Papers with subword-level models
Neural Machine Translation without Embeddings (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models operate over subword tokens, but byte-based models employ a different approach . a one-hot representation of each byte does not hurt performance, but it improves BLEU scores . |
| Approach: | They propose to represent every computerized text as a sequence of bytes via UTF-8 . this eliminates the need for an embedding layer and improves performance . |
| Outcome: | The proposed model improves BLEU scores on byte-to-byte translation models compared to character-level models . the proposed model does not require an embedding layer and does not drop out of the decoder . |
PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing word embeddings assume fixed finite-size vocabularies, hindering their ability to provide useful word representations for out-of-vocaulary words. |
| Approach: | They propose a model that generalizes word embeddings without extra contextual information . they use the spellings of words to model subword segmentation and compute subword-based compositional word embeds. |
| Outcome: | The proposed model can generate meaningful subword segmentations without any source of explicit morphological knowledge. |
An Evaluation of Neural Machine Translation Models on Historical Spelling Normalization (C18-1)
Copied to clipboard
| Challenge: | In this paper, we apply different NMT models to the problem of historical spelling normalization for five languages . we find that NMT model is much better than SMT in terms of character error rate . |
| Approach: | They propose to use NMT models to solve the problem of historical spelling normalization in five languages. |
| Outcome: | The proposed method improves historical spelling normalization for five languages. |
Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora (2023.acl-long)
Copied to clipboard
Svanhvít Lilja Ingólfsdóttir, Petur Ragnarsson, Haukur Jónsson, Haukur Simonarson, Vilhjalmur Thorsteinsson, Vésteinn Snæbjarnarson
| Challenge: | Spelling mistakes due to typos and rushed writing, nonstandard punctuation and spelling, and grammatical and stylistic issues are common to almost everyone who writes any kind of text. |
| Approach: | They propose to use a common subword unit vocabulary and byte-level encoding to fine tune two subword-level models and one byte level model on hand-corrected error corpora. |
| Outcome: | The proposed model improves accuracy for spelling and grammatical errors and more complex errors. |